Multi-cluster warehouse

ABSTRACT

A method for a multi-cluster warehouse includes allocating processing units as part of a data warehouse. The processing units access data within one or more databases in cloud storage resources. The method also includes providing one or more queries to each processing unit within the data warehouse. In response to the queries, each processing unit performs database operations on a particular portion of a database table. The method also includes monitoring a workload of the processing units to determine that a processing capacity of the processing units has reached a threshold processing capacity. The method also includes dynamically adding additional processing units to and removing processing units from the data warehouse as needed based on a workload of the processing units.

PRIORITY

This application is a continuation of U.S. application Ser. No.15/582,071, filed Apr. 28, 2017, which claims the benefit under 35U.S.C. § 119(e) of U.S. Provisional Application No. 62/328,943, filedApr. 28, 2016, entitled “Multi-Cluster Warehouses”, which is herebyincorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure relates to systems, methods, and devices for amulti-cluster warehouse.

BACKGROUND

Databases are widely used for data storage and access in computingapplications. Databases may include one or more tables that include orreference data that can be read, modified, or deleted using queries.Databases can store small or extremely large sets of data within one ormore tables. This data can be accessed by various users in anorganization or even be used to service public users, such as via awebsite or an application program interface (API). Both computing andstorage resources, as well as their underlying architecture, can play alarge role in achieving desirable database performance.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the present disclosureare described with reference to the following figures, wherein likereference numerals refer to like parts throughout the various figuresunless otherwise specified.

FIG. 1 is a block diagram depicting a processing platform according toan example embodiment of the systems and methods described herein.

FIG. 2 is a block diagram illustrating components of a resource manager,according to one embodiment.

FIG. 3 is a block diagram depicting scheduling on a multi-clusterwarehouse, according to one embodiment.

FIG. 4 is a block diagram depicting a plurality of warehouses which maybe provided on a single execution platform, according to one embodiment.

FIG. 5 is a block diagram illustrating a system having multipledistributed virtual warehouses, according to one embodiment.

FIG. 6 is a schematic flow chart diagram illustrating a method for amulti-cluster warehouse, according to one embodiment.

FIG. 7 is a schematic flow chart diagram illustrating a method fordynamically adding compute clusters in a multi-cluster warehouse,according to one embodiment.

FIG. 8 is a schematic flow chart diagram illustrating a method fordynamically removing compute clusters in a multi-cluster warehouse,according to one embodiment.

FIG. 9 is a schematic flow chart diagram illustrating a method for amulti-cluster warehouse, according to one embodiment

FIG. 10 is a block diagram depicting an example computing deviceconsistent with at least one embodiment of processes and systemsdisclosed herein.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present disclosure is directed to system, methods, and devices forproviding and managing multi-cluster warehouses. A warehouse is severalservers that are connected and collaborate in handling analyticalqueries. In some warehouses, compute and storage resources are connectedand allocated together. In at least some embodiments disclosed herein,compute resources are independently allocated and scalable separate fromstorage resources. In some cases, a warehouse includes one or moreclusters and/or a cluster of one or more server nodes that can worktogether to provide services. Applicants have developed, and hereinpresent, architectures, methods, algorithms, and systems formulti-cluster warehouses.

According to one embodiment, a method for a multi-cluster warehouseincludes allocating a plurality of compute clusters as part of a virtualwarehouse. The compute clusters may be used to access and performqueries against one or more databases in one or more cloud storageresources. The method includes providing queries for the virtualwarehouse to each of the plurality of compute clusters. For example,each of the plurality of compute clusters of the virtual warehouse mayreceive a plurality of queries so that the computing load is spreadacross the different clusters. The method may also include dynamicallyadding compute clusters to and removing compute clusters from thevirtual warehouse as needed based on a workload of the plurality ofcompute clusters.

A multi-cluster warehouse can provide significant improvements inconcurrency as well as availability. For example, a warehouse generallyincludes only one single cluster whose size is the size of thewarehouse. For example, a large warehouse may include a single clusterof eight server nodes. A multi-cluster warehouse may allow for creationof a single warehouse with multiple clusters. Each cluster within thewarehouse may include eight server nodes. Thus, the multi-clusterwarehouse may support three times the level of concurrency provided by asingle cluster warehouse of the same size. This architecture can allowfor a high level of concurrency against a single warehouse while alsoallowing for scaling of computing resources, as will be discussedfurther herein.

Improved availability can also be achieved in a multi-cluster warehouseby placing different clusters in different availability zones. Forexample, multi-cluster warehouses will provide improved fault resiliencesince each warehouse cluster could be allocated in a differentavailability zone of a cloud provider (such as within different Amazon®availability zones. Hence, a multi-cluster warehouse would be highlyavailable compare to a single-cluster warehouse. Furthermore, queriescan be routed to an optimal cluster where relevant data segments arealready in memory or local disk-based storage (e.g., in a cache). Forexample, a method for a multi-cluster warehouse may include routingqueries based on a session from which the query originated. By providingqueries from a same session to a same cluster, a likelihood is increasedthat the data needed for a query is already in memory and may eliminatea need to retrieve that data from a cloud storage resource. Withimproved concurrency and availability, users may experience improvedresponse times and availability that would be difficult or impossible toachieve in other traditional single-cluster database architectures.

In addition to improved availability and concurrency, significantvariation in automatic scaling of compute resources is possible. Forexample, at least some embodiments provide separate allocation ofcompute resources from cloud storage. Thus, a multi-cluster warehousecan be scaled up or down in the number of compute clusters toaccommodate wide swings in workload while still querying against thedata that has not changed or is changing very slowly compared to thequery workload.

Automatically resuming or starting a new or suspended cluster may beperformed when the warehouse cannot handle the workload and would haveto queue queries (or queue queries longer than an accepted length oftime). Queries can get queued because the total resource consumption onthe cluster has exceeded a threshold. For example, the resourceconsumption may include parameters for memory load as well as computingor processing load. In one embodiment, a parameter controls for how longa query may be queued before a new cluster should be resumed orprovisioned. As soon as the new cluster has resumed, queries can bescheduled to execute on the new cluster. This applies to new queries aswell as already queued queries.

In one embodiment, a method for a multi-cluster warehouse may includedynamically adding compute clusters to the virtual warehouse based onthe workload. The method may include determining whether a query can beprocessed while meeting a performance metric for the query. If the queryin combination with a current workload does not allow one or morecurrently allocated compute clusters to meet the performance metric, themethod may include triggering startup of a new compute cluster. In someembodiments, a new cluster can be allocated quickly enough to ensurethat not a single query experiences less than the required performancemetric.

Auto-suspending or decommissioning an active cluster of a multi-clusterwarehouse may be performed when the resource consumption of the workloadis low enough that suspending that cluster would not have caused anyquery executed in the past N minutes to be queued (or queued longer thana threshold time). The queueing of a query or a queuing time for a queryis just one example of a performance metric that may be used. In oneembodiment, a method for a multi-cluster warehouse may include removingcompute clusters based on the workload. The method may includedetermining whether a current workload is serviceable by one fewer thanthe plurality of compute clusters while meeting a performance metric.The method may include decommissioning (or suspending) at least onecompute cluster of the plurality of compute clusters in response todetermining that the workload is serviceable by one fewer than theplurality of compute clusters.

According to one embodiment, automatic provisioning or removal ofclusters, as well as routing queries to different clusters within awarehouse, may be used as part of a powerful and flexible multi-clusterwarehouse as a service.

A detailed description of systems and methods consistent withembodiments of the present disclosure is provided below. While severalembodiments are described, it should be understood that this disclosureis not limited to any one embodiment, but instead encompasses numerousalternatives, modifications, and equivalents. In addition, whilenumerous specific details are set forth in the following description inorder to provide a thorough understanding of the embodiments disclosedherein, some embodiments may be practiced without some or all of thesedetails. Moreover, for the purpose of clarity, certain technicalmaterial that is known in the related art has not been described indetail in order to avoid unnecessarily obscuring the disclosure.

Turning to the figures, FIG. 1 is a block diagram illustrating aprocessing platform 100 for providing and/or managing a multi-clusterwarehouse, according to one embodiment. The processing platform 100includes a resource manager 102 that is accessible by multiple users104, 106, and 108. The resource manager 102 may also be referred toherein as a database service manager. In some implementations, resourcemanager 102 can support any number of users desiring access to data orservices of the processing platform 100. Users 104-108 may include, forexample, end users providing data storage and retrieval queries andrequests, system administrators managing the systems and methodsdescribed herein, software applications that interact with a database,and other components/devices that interact with resource manager 102.

The resource manager 102 may provide various services and functions thatsupport the operation of the systems and components within theprocessing platform 100. Resource manager 102 has access to storedmetadata 110 associated with the data stored throughout data processingplatform 100. The resource manager 102 may use the metadata 110 foroptimizing user queries. In some embodiments, metadata 110 includes asummary of data stored in remote data storage systems as well as dataavailable from a local cache (e.g., a cache within one or more of theclusters of the execution platform 112). Additionally, metadata 110 mayinclude information regarding how data is organized in the remote datastorage systems and the local caches. Metadata 110 allows systems andservices to determine whether a piece of data needs to be processedwithout loading or accessing the actual data from a storage device.

As part of the data processing platform 100, metadata 110 may becollected when changes are made to the data using a data manipulationlanguage (DML), which changes may be made by way of any DML statement.Examples of manipulating data may include, but are not limited to,selecting, updating, changing, merging, and inserting data into tables.As part of the processing platform 100, files may be created and themetadata 110 may be collected on a per file and a per column basis,after which the metadata 110 may be saved in a metadata store. Thiscollection of metadata 110 may be performed during data ingestion or thecollection of metadata 110 may be performed as a separate process afterthe data is ingested or loaded. In an implementation, the metadata 110may include a number of distinct values; a number of null values; and aminimum value and a maximum value for each file. In an implementation,the metadata may further include string length information and ranges ofcharacters in strings.

Resource manager 102 is further in communication with an executionplatform 112, which provides multiple computing resources that executevarious data storage and data retrieval operations, as discussed ingreater detail below.

The execution platform 112 may include one or more compute clusterswhich may be dynamically allocated or suspended for specific warehouses,based on the query workload provided by the users 104-108 to a specificwarehouse. The execution platform 112 is in communication with one ormore data storage devices 116, 118, and 120 that are part of a storageplatform 114. Although three data storage devices 116, 118, and 120 areshown in FIG. 1, the execution platform 112 is capable of communicatingwith any number of data storage devices. In some embodiments, datastorage devices 116, 118, and 120 are cloud-based storage deviceslocated in one or more geographic locations. For example, data storagedevices 116, 118, and 120 may be part of a public cloud infrastructureor a private cloud infrastructure, or any other manner of distributedstorage system. Data storage devices 116, 118, and 120 may include harddisk drives (HDDs), solid state drives (SSDs), storage clusters, or anyother data storage technology. Additionally, the storage platform 114may include a distributed file systems (such as Hadoop Distributed FileSystems (HDFS)), object storage systems, and the like.

In some embodiments, the communication links between resource manager102 and users 104-108, metadata 110, and execution platform 112 areimplemented via one or more data communication networks and may beassigned various tasks such that user requests can be optimized.Similarly, the communication links between execution platform 112 anddata storage devices 116-120 in storage platform 114 are implemented viaone or more data communication networks. These data communicationnetworks may utilize any communication protocol and any type ofcommunication medium. In some embodiments, the data communicationnetworks are a combination of two or more data communication networks(or sub-networks) coupled to one another. In alternate embodiments,these communication links are implemented using any type ofcommunication medium and any communication protocol.

As shown in FIG. 1, data storage devices 116, 118, and 120 are decoupledfrom the computing resources associated with execution platform 112.This architecture supports dynamic changes to the data processingplatform 100 based on the changing data storage/retrieval needs,computing needs, as well as the changing needs of the users and systemsaccessing data processing platform 100. The support of dynamic changesallows data processing platform 100 to scale quickly in response tochanging demands on the systems and components within data processingplatform 100. The decoupling of the computing resources from the datastorage devices supports the storage of large amounts of data withoutrequiring a corresponding large amount of computing resources.Similarly, this decoupling of resources supports a significant increasein the computing resources utilized at a particular time withoutrequiring a corresponding increase in the available data storageresources.

The resource manager 102, metadata 110, execution platform 112, andstorage platform 114 are shown in FIG. 1 as individual components.However, each of resource manager 102, metadata 110, execution platform112, and storage platform 114 may be implemented as a distributed system(e.g., distributed across multiple systems/platforms at multiplegeographic locations) or may be combined into one or more systems.Additionally, each of the resource manager 102, storage for metadata110, the execution platform 112, and the storage platform 114 can bescaled up or down (independently of one another) depending on changes tothe requests received from users 104-108 and the changing needs of thedata processing platform 100. Thus, in the described embodiments, thedata processing platform 100 is dynamic and supports regular changes tomeet the current data processing needs.

The execution platform 112 includes a plurality of compute clusters 122,124, 126 which may share a compute or processing load of the processingplatform 100. In one embodiment, customers can control the number ofactive (i.e. running) clusters by specifying a range (e.g., specifyingvalues such as minClusterCount and maxClusterCount) when creating awarehouse or changing its configuration (both while the warehouse isrunning and while it is suspended). Customers may specify an exactnumber of active clusters by specifying, for example, making the minimumcluster count equal to the maximum cluster count so that the warehousewill have that exact number running whenever it is running. If a userspecifies a maximum cluster count that is greater than a minimum clustercount, the resource manager 102 may automatically manage the number ofcurrently active clusters based on the workload to satisfy thethroughput criteria and to be cost-effective. So, whenever the warehouseis running, at least a minimum cluster count (minClusterCount) ofclusters are active, and at most a maximum cluster count(maxClusterCount) of clusters are active. The resource manager 102 maydecide how many clusters are required to handle the current workloadgiven the specified performance criteria in terms of memory load andconcurrency level.

FIG. 2 illustrates a block diagram depicting components of resourcemanager 102, according to one embodiment. The resource manager 102includes an access manager 202 and a key manager 204 coupled to a datastorage device 206. The access manager 202 handles authentication andauthorization tasks for the systems described herein. The key manager204 manages storage and authentication of keys used duringauthentication and authorization tasks. A request processing service 208manages received data storage requests and data retrieval requests. Amanagement console service 210 supports access to various systems andprocesses by administrators and other system managers.

The resource manager 102 also includes an SQL compiler 212, an SQLoptimizer 214 and an SQL executor 216. SQL compiler 212 parses SQLqueries and generates the execution code for the queries. SQL optimizer214 determines the best method to execute queries based on the data thatneeds to be processed. SQL executor 216 executes the query code forqueries received by resource manager 102. A query scheduler andcoordinator 218 sends received queries to the appropriate services orsystems for compilation, optimization, and dispatch to an executionplatform 112. A virtual warehouse manager 220 manages the operation ofmultiple virtual warehouses, including multi-cluster warehouses,implemented in execution platform 112.

Additionally, the resource manager 102 includes a configuration andmetadata manager 222, which manages the information related to the datastored in the remote data storage devices and in the local caches. Amonitor and workload analyzer 224 oversees the processes performed bythe resource manager 102 and manages the distribution of tasks (e.g.,workload) across the virtual warehouses and execution nodes in theexecution platform 112. Configuration and metadata manager 222 andmonitor and workload analyzer 224 are coupled to a data storage device226.

The resource manager 102 also includes a transaction management andaccess control module 228, which manages the various tasks and otheractivities associated with the processing of data storage requests anddata access requests. For example, the transaction management and accesscontrol module 228 provides consistent and synchronized access to databy multiple users or systems. Since multiple users/systems may accessthe same data simultaneously, changes to the data must be synchronizedto ensure that each user/system is working with the current version ofthe data. Transaction management and access control module 228 providescontrol of various data processing activities at a single, centralizedlocation in resource manager 102.

With further reference to the virtual warehouse manager 220, automaticcluster resume and automatic cluster suspend in a multi-clusterwarehouse will be discussed. In one embodiment, the virtual warehousemanager 220 will perform automatic cluster resume. When a multi-clusterwarehouse (e.g., within the execution platform 112) is marked forautomatic resume, the first cluster for the warehouse will beautomatically resumed when a SQL statement is scheduled and all clustersin the warehouse are in a suspended state. But the decision toautomatically resume the remaining clusters will be performed based onthe workload. This assumes that activeClusterCount<maxClusterCount, thatis, we have clusters that may be activated/resumed, but are currentlysuspended.

Workload considerations include at least two things. First, workloadconsiderations may account for memory usage. When queries are scheduledand are queued because all clusters are at their maximum memorycapacity, the virtual warehouse manager 220 will resume one or moreclusters so that queueing can be avoided, or shortened. Queuing maystill occur if new clusters need to be resumed since resuming a clustermay take a bit of time, for example in minutes. However, the virtualwarehouse manager 220 may also make sure that there is a free pool ofseveral free servers so that queries can be put on the free pool duringthe starting of the new cluster. Also, the virtual warehouse manager 220may wait a specific period of time to see if queuing will resolve byitself before deciding to provision a new cluster.

Second, workload considerations may account for a degree of concurrence,or the processing/computing load on a cluster. If the degree ofconcurrency is high on all active clusters, then the virtual warehousemanager 220 may start another cluster even if there is enough memory toschedule the query. Here, the degree of concurrency may be computed foreach active cluster based on the degree of parallelism (DOP).Specifically, the degree of concurrency may be the number of queriesrunning at full DOP. For example, this may be calculated as the sum theDOP for all running queries divided by the product of a max DOP(MAX_DOP) and the number of running queries. This can be fractional ornon-integer value since some lightweight queries are running with asmaller DOP than the max. In one embodiment, a warehouse parameter maybe specified to control the degree of concurrency. For example, thedegree of concurrency (concurrency_level_target_may be set to the valueof 8 by default. This parameter may be exposed to a customer since itsvalue really depends on how much money the customer wants to put on thatproblem and how much they are willing to allow query performance todegrade when a warehouse is shared (compared to the stand-alone queryperformance).

In one embodiment, the virtual warehouse manager 220 will performautomatic cluster suspend. In one embodiment, the full warehouse mayshut down after a specific number of seconds (e.g., based on anauto_suspend parameter) of inactivity. Orthogonally to this, when thewarehouse has more than one active cluster, one or more clusters may besuspended if the warehouse was running below its capacity for more thana specified amount of time, e.g. measured in minutes. For example,consider a warehouse with three active clusters. If for more than aspecified time period the warehouse is under-loaded, i.e. would havebeen able to execute all SQL statements issued in the specified timeperiod engine at the current time without any queueing or without goingover the maximum degree of concurrency, then one or more clusters wouldbe suspended. Note that while a warehouse is underloaded, it will stillleverage all clusters currently active. A cluster does not need to beinactive for a specific number of minutes to be shut down. A check forautomatic cluster suspend may be performed on a periodic basis, such asfor the last 5 minutes, last 10 minutes, last half hour, last hour, etc.

In one embodiment, the check for automatic cluster may be performed atan interval different than the specified time period. For example, thecheck whether the last 10 minutes have been below load may be performedupon each hour change so that a customer can be charged on an hourlybasis.

With further reference to the query scheduler and coordinator 218, queryscheduling may be performed based on workload, query affinity, and otherfactors. The query scheduler and coordinator 218 and may forward queriesto a specific cluster based on workload. For example, the queryscheduler and coordinator 218 may attempt to maintain an approximatelyequal workload on each cluster to spread out processing tasks and toimprove query execution time and user experience. Query affinity may beused so that related queries, especially queries related to the samedata, will be sent to the same cluster. For example, the query schedulerand coordinator 218 may send queries having a same session identifier tothe same cluster. Forwarding queries based on query affinity may allowthe query scheduler and coordinator 218 to ensure that the data againstwhich a query is to be performed is already in the local cache of aspecific cluster. This can significantly reduce response time, workload,and data lookup.

FIG. 3 is a schematic block diagram illustrating a multi-clusterwarehouse 302 and the scheduling of queries 304 on the multi-clusterwarehouse 302. The warehouse 302 includes a plurality of clusters(Cluster 1, Cluster 2, Cluster N) that each include a plurality ofserver nodes. In one embodiment, each of the clusters includes the samenumber of servers although this may be different in differentembodiments. In one embodiment, each server in a cluster belong to thesame availability zone but different clusters may be placed in differentavailability zones. The concept of availability of the warehouse may bebased on overall availability percentage of the warehouse. For example,the availability for a specific cluster within the warehouse 302 may bethe percentage of servers which are available (e.g., in an operationalstate) relatively to the cluster size. However, when that percentagegoes below the minimum (e.g., 50%) required to run a query 0%availability may be determined for that cluster and no queries may beassigned until the warehouse 302, or some of the servers in thewarehouse 302, is repaired. As discussed herein, the number of clustersin the warehouse 302 may be adjusted dynamically based on workload,server failures in the clusters, or the like.

In one embodiment, the query scheduler and coordinator 218 weights eachquery (e.g., SQL statement or portion of a SQL statement) based on itsprojected resource consumption. For example, some queries may takesignificantly more memory to perform while other queries may takesignificantly more processing resources to perform. Similarly, somequeries may have high or low consumption for both memory and processing.The resource manager 102 may determine what the predicted or projectedconsumption is and then may be able to determine where to place thequery to most efficiently balance the workload among different clusters.For example, on high consumption query may use as many resources asmultiple low consumption queries.

In one embodiment, the query scheduler and coordinator 218 may schedulequeries on the one or more clusters of the warehouse 302 or may queuequeries when workload is too high or availability is too low. Forexample, the query scheduler and coordinator 218 may first attempt toschedule a query 304 (e.g. a SQL statement) on an active (i.e. notsuspended) cluster of the warehouse 302. If there are multiple activeclusters, the query scheduler and coordinator 218 will eliminate the setof clusters which are not available or where the query 304 would not runbecause memory would be over-subscribed. As mentioned previously, acluster may be determined not available by default if less than 50% ofthe servers of a node are not available (e.g., have failed). If thereare multiple possible clusters remaining, the query scheduler andcoordinator 218 may then pick the least loaded cluster. The least loadedcluster, in one embodiment, is defined as the sum of the DOPs of alljobs running on that cluster. The least loaded cluster may also be basedon the sum of all memory requirements for that cluster. If there aremultiple clusters with equal load, the query scheduler and coordinator218 may use the session ID for the specific query 304 as a tie-breakersuch that queries from the same session can execute on the same cluster.Queries 304 that have been assigned to a cluster in the warehouse 302are shown as running queries 306.

If there are not any clusters to schedule a specific query, then thequery scheduler and coordinator 218 may queue the query in a globalqueue. Globally queued queries 304 are shown as queued queries 308.Queued queries 308 may remain queued until one of the cluster of thewarehouse 302 is freed up or becomes available. Note that one or moreservers in an assigned cluster might be marked as suspected failed inwhich case some running queries 306 may also have to be queued waitingfor the cluster to be repaired.

FIG. 4 is a block diagram depicting an embodiment of a plurality ofwarehouses which may be active or operating on a single executionplatform 112, according to one embodiment. Multiple virtual warehouses402, 404, 406 are shown and each virtual warehouse includes a pluralityof clusters 408. Each cluster 408 includes multiple execution nodes 410that each include a processor 412 and a cache 414 (e.g. memory).Although three virtual warehouses 402-406 are shown, the number ofvirtual warehouses may change dynamically. Similarly, the number ofclusters 408 in each warehouse 402-406, and the number of executionnodes 410 in each cluster may vary in different embodiments and may alsovary relative to each other without limitation. Furthermore, the numberof clusters 408 in a virtual warehouse and a number of execution nodes410 in a cluster may be dynamic, such that new clusters 408 andexecution nodes 410 are created or removed when demand changes.

Each virtual warehouse 402-406 is capable of accessing any of the datastorage devices 116-120 shown in FIG. 1. Thus, virtual warehouses402-406 are not necessarily assigned to a specific data storage device116-120 and, instead, can access data from any of the data storagedevices 116-120. Similarly, each of the clusters 408 and execution nodes410 can access data from any of the data storage devices 116-120. Insome embodiments, a particular virtual warehouse or a particularexecution node may be temporarily assigned to a specific data storagedevice, but the virtual warehouse or execution node may later accessdata from any other data storage device.

Although the illustrated execution nodes 410 each include one cache andone processor, alternate embodiments may include execution nodescontaining any number of processors and any number of caches.Additionally, the caches may vary in size among the different executionnodes 410. The caches 414 store, in the local execution node, data thatwas retrieved from one or more data storage devices in storage platform114 (FIG. 1). Thus, the caches reduce or eliminate the bottleneckproblems occurring in platforms that consistently retrieve data fromremote storage systems. Instead of repeatedly accessing data from theremote storage devices, the systems and methods described herein accessdata from the caches in the execution nodes which is significantlyfaster and avoids the bottleneck problem discussed above. In someembodiments, the caches are implemented using high-speed memory devicesthat provide fast access to the cached data. Each cache can store datafrom any of the storage devices in storage platform 114.

Further, the cache resources and computing resources may vary betweendifferent execution nodes. For example, one execution node may containsignificant computing resources and minimal cache resources, making theexecution node useful for tasks that require significant computingresources. Another execution node may contain significant cacheresources and minimal computing resources, making this execution nodeuseful for tasks that require caching of large amounts of data. In someembodiments, the cache resources and computing resources associated witha particular execution node are determined when the execution node iscreated, based on the expected tasks to be performed by the executionnode.

Additionally, the cache resources and computing resources associatedwith a particular execution node may change over time based on changingtasks performed by the execution node. For example, a particularexecution node may be assigned more processing resources if the tasksperformed by the execution node become more processor intensive.Similarly, an execution node may be assigned more cache resources if thetasks performed by the execution node require a larger cache capacity.

Although virtual warehouses 402-406 are associated with the sameexecution platform 112, the virtual warehouses may be implemented usingmultiple computing systems at multiple geographic locations. Forexample, virtual warehouse 402 can be implemented by a computing systemat a first geographic location, while virtual warehouses 404 and 406 areimplemented by another computing system at a second geographic location.In some embodiments, these different computing systems are cloud-basedcomputing systems maintained by one or more different entities.

Additionally, each virtual warehouse is shown in FIG. 4 as havingmultiple clusters 408. The clusters 408 associated with each virtualwarehouse may be implemented using multiple computing systems atmultiple geographic locations or within different availability zones.For example, a particular instance of virtual warehouse 402 implementsclusters 408 with execution nodes 410 on one computing platform at aparticular geographic location, and implements other clusters 408 andexecution nodes 410 at a different computing platform at anothergeographic location. The virtual warehouses 402-406 are also faulttolerant. For example, if one virtual warehouse or an execution node410, that virtual warehouse or execution node is quickly replaced at thesame or different geographic location.

A particular execution platform 112 may include any number of virtualwarehouses 402-406. Additionally, the number of virtual warehouses in aparticular execution platform is dynamic, such that new virtualwarehouses are created when additional processing and/or cachingresources are needed. Similarly, existing virtual warehouses may bedeleted when the resources associated with the virtual warehouse are nolonger necessary.

FIG. 5 illustrates a block diagram depicting another example operatingenvironment 500 having multiple distributed virtual warehouses andexecution platform groups. Environment 500 includes resource manager 102that communicates with execution platform group 1 504 and executionplatform group 2 506 through a data communication network 502. Executionplatform group 1 504 includes two clusters, specifically, cluster A fora first virtual warehouse 508 and cluster A for a second virtualwarehouse 510. Execution platform group 2 506 includes two additionalclusters, specifically, cluster B for the first virtual warehouse 514and cluster B for the second virtual warehouse 516. The resource manager102 also communicates with cluster C of the first virtual warehouse 512(which is not part of either of the execution platform groups 504, 506)through data communication network 502.

Execution platform groups 504 and 506 as well as cluster C for the firstvirtual warehouse 512 communicate with databases 520, 522, and 524through a data communication network 518. In some embodiments datacommunication networks 502 and 518 are the same network or a combinationof one or more overlapping networks. Environment 500 allows resourcemanager 102 to coordinate user data storage and retrieval requestsacross multiple clusters 508-516 of multiple warehouses to store andretrieve data in databases 520-524. Execution platform groups 504 and506, as well as cluster C for the first virtual warehouse 512, can belocated in the same or different geographic area, or can be located inthe same or different availability zones. Additionally, executionplatform groups 504 and 506 can be implemented by the same entity or bydifferent entities.

The systems and methods described herein allow data to be stored andaccessed as a service that is separate from computing (or processing)resources. Even if no computing resources have been requested from theexecution platform, data is available to a virtual warehouse withoutrequiring reloading of the data from a remote data source. The describedsystems and methods are useful with any type of data. In particularembodiments, data is stored in a structured, optimized format. Thedecoupling of the data storage/access service from the computingservices also simplifies the sharing of data among different users andgroups. As discussed herein, each virtual warehouse can access any datato which it has access permissions, even at the same time as othervirtual warehouses are accessing the same data. This architecturesupports running queries without any actual data stored in the localcache. The systems and methods described herein are capable oftransparent dynamic data movement, which moves data from a remotestorage device to a local cache, as needed, in a manner that istransparent to the user of the system. Further, this architecturesupports data sharing without prior data movement since any virtualwarehouse can access any data due to the decoupling of the data storageservice from the computing service.

Furthermore, the environment 500 allows for the spreading of a singlevirtual warehouse across multiple geographic locations or availabilityzones. For example, clusters 508, 512 and 514 all belong to the samevirtual warehouse (the first virtual warehouse) but may be located indifferent geographic areas or availability zones. Because outages orfailures may happen across a geographic area or availability zone,improved fault tolerance may be achieved. For example, availabilityzones are sometimes implemented by cloud service (compute or storage)providers so that problems in one availability zone have little or nochance of propagating to a different availability zone. Thus, clusterswithin the same warehouse but in different availability zones cansignificantly decrease the likelihood that a warehouse is left withoutany available execution or compute nodes.

In one embodiment, the multi-cluster warehouse embodiments disclosedherein may use a special data definition language (DDL). The followingare some examples of commands or instructions which may be part of amulti-cluster warehouse DDL:

-   -   create warehouse single_cluster size=xlarge;    -   //this will create a single_cluster warehouse    -   create warehouse multi_cluster size=xlarge max_cluster_count=3    -   min_cluster_count=1;    -   //this will create an x-large 3 cluster warehouse. Only one        cluster will be started by default    -   create warehouse multi_cluster size=xlarge max_cluster_count=3    -   min_cluster_count=2;    -   //this will create an x-large warehouse with 2 clusters        initially resumed    -   create warehouse multi_cluster size=xlarge max_cluster_count=3    -   min_cluster_count=3;    -   //this will create an x-large warehouse with all clusters        resumed    -   Note that the resource manager would try to make use of all        availability zones, one per cluster. The availability zone to        use for each cluster may be implemented by an infrastructure        management system    -   alter warehouse <warehouse_name>set warehouse_size=<size>:        allows one to change the size of the warehouse. If this        warehouse is started, all clusters in the warehouse will be        resized. The code to implement this instruction may include a        resize operation for each cluster.    -   alter warehouse <warehouse_name>set max_cluster_count=<count>:        this will add or remove clusters from an existing warehouse.        Internally clusters may be numbered so this operation will        either add new clusters at the end of the range or remove        clusters starting from the end of the range. If new clusters are        created, they will be created in a suspended state. If clusters        are removed and these clusters are active, they will first be        inactivated (quiesced) to allow running queries to terminate.    -   drop warehouse <warehouse_name>: drop warehouse and all        associated clusters. Clusters will be inactivated (quiesced)        before dropping them.

Returning to the figures, FIG. 6 is a schematic flow chart diagramillustrating an example method 600 for a multi-cluster warehouse. Themethod 600 may be performed by a processing platform or a resourcemanager, such as the processing platform 100 of FIG. 1 or the resourcemanager of FIG. 1, 2, or 5.

The method 600 begins and a system allocates 602 a plurality of computeclusters as part of a virtual warehouse for accessing and performingqueries against one or more databases in one or more cloud storageresources. In one embodiment, the plurality of compute clusters isallocated by the system independently from the one or more cloud storageresources such that the number of compute clusters can be scaled up anddown without increasing or decreasing the one or more cloud storageresources. The system provides 604 queries for the virtual warehouse toeach of the plurality of compute clusters. For example, the plurality ofqueries may be provided to each of the plurality of compute clusters ofthe virtual warehouse. The system dynamically adds 606 compute clustersto and removes compute clusters from the virtual warehouse as neededbased on a workload of the plurality of compute clusters. The method 600may also include determining the workload for the plurality of computeclusters. The system may determine the workload by determining anavailability of one or more of processor resources for each of theplurality of compute clusters and memory resources for each of theplurality of compute clusters.

Method 600 may be implemented by a database system or device to allow asingle entity, such as a warehouse, to expand and shrink depending on anumber of queries. Specifically, as changes in the concurrency (orcompute and memory load) of a warehouse occur, a resource manager orother system may allow the warehouse to scale up and down

FIG. 7 is a schematic flow chart diagram illustrating an example method700 for dynamically adding compute clusters in a multi-clusterwarehouse. The method 700 may be performed by a processing platform or aresource manager, such as the processing platform 100 of FIG. 1 or theresource manager of FIG. 1, 2, or 5. The method 700 may be performed inconjunction with or separately from method 600 of FIG. 6.

The method 700 begins and a system determines 702 whether a query can beprocessed while meeting a performance metric for the query. In oneembodiment, the method 700 includes determining 702 whether the querycan be processed for each query directed to the compute cluster suchthat the performance metric is met for each query. The performancemetric may include a service level agreement (SLA) accepted by acustomer. For example, the SLA may require that a query be scheduledwithin a specific amount of time (e.g., 10 seconds). This may restrictany query from being queued in a global queue longer than a maximum time(e.g., 10 seconds). The SLA may be agreed to in advance between awarehouse as a service provider and a customer. Different price tiersmay be presented based on what the SLA is, or the SLA may dictate thatthe system uses more resources to ensure that users experience minimumdelay in accessing and performing queries against a database.

The system triggers 704 startup of a new compute cluster in response todetermining that the query in combination with a current workload doesnot allow one or more currently allocated compute clusters to meet theperformance metric. In one embodiment, the system may only trigger 704startup if the number of currently active clusters is less than apredetermined maximum number of compute clusters

FIG. 8 is a schematic flow chart diagram illustrating an example method800 for dynamically removing compute clusters in a multi-clusterwarehouse. The method 800 may be performed by a processing platform or aresource manager, such as the processing platform 100 of FIG. 1 or theresource manager of FIG. 1, 2, or 5. The method 800 may be performed inconjunction with or separately from one or more of methods 600 or 700 ofFIGS. 6 and 7.

The method 800 begins and a system determines 802 whether a currentworkload is serviceable by one fewer than the plurality of computeclusters while meeting a performance metric. In one embodiment,determining 802 whether the current workload is serviceable by one fewerthan the plurality of compute clusters may include determining whether ahistorical workload for a time period leading up to the current time wasserviceable by one fewer than the plurality of clusters while meetingthe performance metric. For example, if the best cluster were removedfrom the virtual warehouse, would the virtual warehouse have been ableto process all the queries while meeting the performance metric?

The system decommissions 804 (or inactivating) at least one computecluster of the plurality of compute clusters in response to determiningthat the workload is serviceable by one fewer than the plurality ofcompute clusters. The system may only decommission 804 or remove acompute cluster if the current number of active clusters is less than apredetermined minimum number of compute clusters. In one embodiment,decommissioning 804 the at least one compute cluster may includedecommissioning in response to determining that the historical workloadfor the time period was serviceable by one fewer than the plurality ofcompute clusters.

In one embodiment, decommissioning 804 the at least one compute clusterincludes making a cluster quiescent to prevent providing or schedulingof additional queries to the at least one compute cluster.Decommissioning 804 may also include allowing the at least one computecluster to complete currently assigned queries and releasing one or moreresources corresponding to the at least one compute cluster uponcompletion of the already scheduled or active queries.

FIG. 9 is a schematic flow chart diagram illustrating an example method900 for a multi-cluster warehouse. The method 900 may be performed by aprocessing platform or a resource manager, such as the processingplatform 100 of FIG. 1 or the resource manager of FIG. 1, 2, or 5. Themethod 900 may be performed in conjunction with or separately from oneor more of methods 600, 700, or 800 of FIGS. 6, 7, and 8.

The method 900 begins and a system allocates 902 a plurality of computeclusters as part of a virtual warehouse for accessing and performingqueries against one or more databases in one or more cloud storageresources. The system forwards 904 queries for the virtual warehouse toeach of the plurality of compute clusters. The plurality of queries maybe provided to each of the plurality of compute clusters of the virtualwarehouse. In one embodiment, forwarding 904 queries for the virtualwarehouse to each of the plurality of compute clusters includes routing906 queries based on a session from which the query originated, suchthat queries from the same session are routed to a same compute clusterby default. Each cluster has the ability to persist fragments of thedatabase it is operating on. That is, each cluster (or each compute nodein the cluster) may maintain a cache of all the tables that it hascurrently accessed while processing queries on a cluster. Thus, theresource manager or scheduler drives the queries from the same querystream (e.g., having the same session identifier) to the same cluster sothey can leverage the caching effect. In some cases, if a cluster thatis handling a specific session has much less available resources thananother cluster, queries with the same session identifier may end up ondifferent clusters.

In one embodiment, the system may route 906 the queries based on aworkload of each of the plurality of compute clusters. For example, if acluster cannot accept new queries, the system may provide the query to adifferent cluster even if the different cluster has not processed queuescorresponding to the same session. In one embodiment, the system mayprovide 904 queries to at least two compute clusters in differentavailability zones.

FIG. 10 is a block diagram depicting an example computing device 1000.In some embodiments, computing device 1000 is used to implement one ormore of the systems and components discussed herein. For example,computing device 1000 may allow a user or administrator to accessresource manager 102. As another example, the components, systems, orplatforms discussed herein may include one or more computing devices1000. Further, computing device 1000 may interact with any of thesystems and components described herein. Accordingly, computing device1000 may be used to perform various procedures and tasks, such as thosediscussed herein. Computing device 1000 can function as a server, aclient or any other computing entity. Computing device 1000 can be anyof a wide variety of computing devices, such as a desktop computer, anotebook computer, a server computer, a handheld computer, a tablet, andthe like.

Computing device 1000 includes one or more processor(s) 1002, one ormore memory device(s) 1004, one or more interface(s) 1006, one or moremass storage device(s) 1008, and one or more Input/Output (I/O)device(s) 1010, all of which are coupled to a bus 1012. Processor(s)1002 include one or more processors or controllers that executeinstructions stored in memory device(s) 1004 and/or mass storagedevice(s) 1008. Processor(s) 1002 may also include various types ofcomputer-readable media, such as cache memory.

Memory device(s) 1004 include various computer-readable media, such asvolatile memory (e.g., random access memory (RAM)) and/or nonvolatilememory (e.g., read-only memory (ROM)). Memory device(s) 1004 may alsoinclude rewritable ROM, such as Flash memory.

Mass storage device(s) 1008 include various computer readable media,such as magnetic tapes, magnetic disks, optical disks, solid statememory (e.g., Flash memory), and so forth. Various drives may also beincluded in mass storage device(s) 1008 to enable reading from and/orwriting to the various computer readable media. Mass storage device(s)1008 include removable media and/or non-removable media.

I/O device(s) 1010 include various devices that allow data and/or otherinformation to be input to or retrieved from computing device 1000.Example I/O device(s) 1010 include cursor control devices, keyboards,keypads, microphones, monitors or other display devices, speakers,printers, network interface cards, modems, lenses, CCDs or other imagecapture devices, and the like.

Interface(s) 1006 include various interfaces that allow computing device1000 to interact with other systems, devices, or computing environments.Example interface(s) 1006 include any number of different networkinterfaces, such as interfaces to local area networks (LANs), wide areanetworks (WANs), wireless networks, and the Internet.

Bus 1012 allows processor(s) 1002, memory device(s) 1004, interface(s)1006, mass storage device(s) 1008, and I/O device(s) 1010 to communicatewith one another, as well as other devices or components coupled to bus1012. Bus 1012 represents one or more of several types of busstructures, such as a system bus, PCI bus, IEEE 1394 bus, USB bus, andso forth.

Examples

The following examples pertain to further embodiments.

Example 1 is a computer implemented method for a multi-clusterwarehouse. The method includes allocating a plurality of computeclusters as part of a virtual warehouse for accessing and performingqueries against one or more databases in one or more cloud storageresources. The method includes providing queries for the virtualwarehouse to each of the plurality of compute clusters, wherein aplurality of queries is provided to each of the plurality of computeclusters of the virtual warehouse. The method includes dynamicallyadding compute clusters to and removing compute clusters from thevirtual warehouse as needed based on a workload of the plurality ofcompute clusters.

In Example 2, the plurality of compute clusters of Example 1 isallocated independently from the one or more cloud storage resourcessuch that the number of compute clusters can be scaled up and downwithout increasing or decreasing the one or more cloud storageresources.

In Example 3, the method in any of Examples 1-2 further includesdetermining the workload for the plurality of compute clusters.Determining the workload includes determining an availability of one ormore of processor resources for each of the plurality of computeclusters memory resources for each of the plurality of compute clusters.

In Example 4, the dynamically adding compute clusters in any of Examples1-3 includes determining whether a query can be processed while meetinga performance metric for the query and triggering startup of a newcompute cluster in response to determining that the query in combinationwith a current workload does not allow one or more currently allocatedcompute clusters to meet the performance metric.

In Example 5, the method of Example 4 includes determining whether thequery can be processed for each query directed to the compute clustersuch that the performance metric is met for each query.

In Example 6, the performance metric in any of Examples 4-5 includes aservice level agreement accepted by a customer.

In Example 7, the performance metric in any of Examples 4-6 includes amaximum time period that the query will be queued.

In Example 8, the dynamically adding compute clusters in any of Examples1-7 includes adding compute clusters up to a predetermined maximumnumber of compute clusters.

In Example 9, the dynamically removing compute clusters in any ofExamples 1-8 includes removing compute clusters down to a predeterminedminimum number of compute clusters.

In Example 10, the removing compute clusters in any of Examples 1-9includes determining whether a current workload is serviceable by onefewer than the plurality of compute clusters while meeting a performancemetric and decommissioning at least one compute cluster of the pluralityof compute clusters in response to determining that the workload isserviceable by one fewer than the plurality of compute clusters.

In Example 11, the decommissioning the at least one compute cluster inExample 10 includes: preventing providing additional queries to the atleast one compute cluster; allowing the at least one compute cluster tocomplete currently assigned queries; and releasing one or more resourcescorresponding to the at least one compute cluster upon completion of thecurrently assigned queries.

In Example 12, the determining whether the current workload isserviceable by one fewer than the plurality of compute clusters in anyof Examples 10-11 includes determining whether a historical workload fora time period leading up to the current time was serviceable by onefewer than the plurality of clusters while meeting the performancemetric. Decommissioning the at least one compute cluster includesdecommissioning in response to determining that the historical workloadfor the time period was serviceable by one fewer than the plurality ofcompute clusters.

In Example 13, the providing queries for the virtual warehouse to eachof the plurality of compute clusters in any of Examples 1-12 includesrouting queries based on a session from which the query originated.

In Example 14, the providing queries for the virtual warehouse to eachof the plurality of compute clusters in any of Examples 1-13 includesrouting queries based on a workload of each of the plurality of computeclusters.

In Example 15, the allocating the plurality of compute clusters in anyof Examples 1-14 allocating at least two compute clusters in differentavailability zones

Example 16 is an apparatus including means to perform a method as in anyof Examples 1-15.

Example 17 is a machine-readable storage including machine-readableinstructions that, when executed, implement a method or realize anapparatus of any of Examples 1-16.

The flow diagrams and block diagrams herein illustrate the architecture,functionality, and operation of possible implementations of systems,methods, and computer program products according to various embodimentsof the present disclosure. In this regard, each block in the flowdiagrams or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It will also be notedthat each block of the block diagrams and/or flow diagrams, andcombinations of blocks in the block diagrams and/or flow diagrams, maybe implemented by special purpose hardware-based systems that performthe specified functions or acts, or combinations of special purposehardware and computer instructions. These computer program instructionsmay also be stored in a computer-readable medium that can direct acomputer or other programmable data processing apparatus to function ina particular manner, such that the instructions stored in thecomputer-readable medium produce an article of manufacture includinginstruction means which implement the function/act specified in the flowdiagram and/or block diagram block or blocks.

The systems and methods described herein provide a flexible and scalabledata warehouse using new data processing platforms, methods, systems,and algorithms. In some embodiments, the described systems and methodsleverage a cloud infrastructure that supports cloud-based storageresources, computing resources, and the like. Example cloud-basedstorage resources offer significant storage capacity available on-demandat a low cost. Further, these cloud-based storage resources may befault-tolerant and highly scalable, which can be costly to achieve inprivate data storage systems. Example cloud-based computing resourcesare available on-demand and may be priced based on actual usage levelsof the resources. Typically, the cloud infrastructure is dynamicallydeployed, reconfigured, and decommissioned in a rapid manner.

In the described systems and methods, a data storage system utilizes anSQL (Structured Query Language)-based relational database. However,these systems and methods are applicable to any type of database usingany data storage architecture and using any language to store andretrieve data within the database. The systems and methods describedherein may also provide a multi-tenant system that supports isolation ofcomputing resources and data between different customers/clients andbetween different users within the same customer/client.

Various techniques, or certain aspects or portions thereof, may take theform of program code (i.e., instructions) embodied in tangible media,such as floppy diskettes, CD-ROMs, hard drives, a non-transitorycomputer readable storage medium, or any other machine readable storagemedium wherein, when the program code is loaded into and executed by amachine, such as a computer, the machine becomes an apparatus forpracticing the various techniques. In the case of program code executionon programmable computers, the computing device may include a processor,a storage medium readable by the processor (including volatile andnon-volatile memory and/or storage elements), at least one input device,and at least one output device. The volatile and non-volatile memoryand/or storage elements may be a RAM, an EPROM, a flash drive, anoptical drive, a magnetic hard drive, or another medium for storingelectronic data. One or more programs that may implement or utilize thevarious techniques described herein may use an application programminginterface (API), reusable controls, and the like. Such programs may beimplemented in a high-level procedural or an object-oriented programminglanguage to communicate with a computer system. However, the program(s)may be implemented in assembly or machine language, if desired. In anycase, the language may be a compiled or interpreted language, andcombined with hardware implementations.

It should be understood that many of the functional units described inthis specification may be implemented as one or more components, whichis a term used to more particularly emphasize their implementationindependence. For example, a component may be implemented as a hardwarecircuit comprising custom very large scale integration (VLSI) circuitsor gate arrays, off-the-shelf semiconductors such as logic chips,transistors, or other discrete components. A component may also beimplemented in programmable hardware devices such as field programmablegate arrays, programmable array logic, programmable logic devices, orthe like.

Components may also be implemented in software for execution by varioustypes of processors. An identified component of executable code may, forinstance, comprise one or more physical or logical blocks of computerinstructions, which may, for instance, be organized as an object, aprocedure, or a function. Nevertheless, the executables of an identifiedcomponent need not be physically located together, but may comprisedisparate instructions stored in different locations that, when joinedlogically together, comprise the component and achieve the statedpurpose for the component.

Indeed, a component of executable code may be a single instruction, ormany instructions, and may even be distributed over several differentcode segments, among different programs, and across several memorydevices. Similarly, operational data may be identified and illustratedherein within components, and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set, or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork. The components may be passive or active, including agentsoperable to perform desired functions.

Reference throughout this specification to “an example” means that aparticular feature, structure, or characteristic described in connectionwith the example is included in at least one embodiment of the presentdisclosure. Thus, appearances of the phrase “in an example” in variousplaces throughout this specification are not necessarily all referringto the same embodiment.

As used herein, a plurality of items, structural elements, compositionalelements, and/or materials may be presented in a common list forconvenience. However, these lists should be construed as though eachmember of the list is individually identified as a separate and uniquemember. Thus, no individual member of such list should be construed as ade facto equivalent of any other member of the same list solely based onits presentation in a common group without indications to the contrary.In addition, various embodiments and examples of the present disclosuremay be referred to herein along with alternatives for the variouscomponents thereof. It is understood that such embodiments, examples,and alternatives are not to be construed as de facto equivalents of oneanother, but are to be considered as separate and autonomousrepresentations of the present disclosure.

Although the foregoing has been described in some detail for purposes ofclarity, it will be apparent that certain changes and modifications maybe made without departing from the principles thereof. It should benoted that there are many alternative ways of implementing both theprocesses and apparatuses described herein. Accordingly, the presentembodiments are to be considered illustrative and not restrictive.

Those having skill in the art will appreciate that many changes may bemade to the details of the above-described embodiments without departingfrom the underlying principles of the disclosure. The scope of thepresent disclosure should, therefore, be determined only by thefollowing claims.

What is claimed is:
 1. A method, comprising: allocating a plurality ofprocessing units as part of a data warehouse, wherein each of at leastsome of the plurality of processing units accesses data within one ormore databases in one or more cloud storage resources; providing, by aprocessing device, one or more queries to each processing unit withinthe data warehouse; monitoring a workload of the plurality of theprocessing units to determine that a processing capacity of theplurality of processing units has reached a threshold processingcapacity; and dynamically adding additional processing units to andremoving processing units from the data warehouse as needed based on theworkload of the processing units indicating that the processing capacityhas reached the threshold processing capacity.
 2. The method of claim 1,wherein determining the processing capacity of the plurality ofprocessing units comprises determining an availability of one or moreof: processor resources for each processing unit; and memory resourcesfor each processing unit.
 3. The method of claim 1, wherein dynamicallyadding processing units to the data warehouse based on the workloadcomprises: determining whether a query can be processed while meeting aperformance metric for the query; and triggering startup of a newprocessing unit in response to determining that the query in combinationwith a current workload does not allow one or more currently allocatedprocessing units to meet the performance metric.
 4. The method of claim3, wherein the method comprises determining whether the query can beprocessed for each query directed to the processing unit such that theperformance metric is met for each query.
 5. The method of claim 3,wherein the performance metric comprises a maximum time period that thequery will be queued.
 6. The method of claim 1, wherein dynamicallyadding processing units comprises adding processing units up to apredetermined maximum number of processing units.
 7. The method of claim1, wherein dynamically removing processing units comprises removingprocessing units down to a predetermined minimum number of processingunits.
 8. The method of claim 1, wherein removing processing units basedon the workload comprises: determining whether a current workload isserviceable by one fewer than the plurality of processing units whilemeeting a performance metric; and decommissioning at least oneprocessing unit of the plurality of processing units in response todetermining that the workload is serviceable by one fewer than theplurality of processing units.
 9. The method of claim 1, whereinproviding queries for the data warehouse to each of the plurality ofprocessing units comprises routing queries based on a workload of eachof the plurality of processing units.
 10. A system, comprising: aprocessing device programmed to: allocate a plurality of processingunits as part of a data warehouse, wherein each of at least some of theplurality of processing units accesses data within one or more databasesin one or more cloud storage resources; provide, by a processing device,one or more queries to each processing unit within the data warehouse;monitor a workload of the plurality of the processing units to determinethat a processing capacity of the plurality of processing units hasreached a threshold processing capacity; and dynamically add additionalprocessing units to and removing processing units from the datawarehouse as needed based on the workload of the processing unitsindicating that the processing capacity has reached the thresholdprocessing capacity.
 11. The system of claim 10, further comprisingdetermining the workload of the processing units, wherein determiningthe workload comprises determining an availability of one or more of:processor resources for each processing unit; and memory resources foreach processing unit.
 12. The system of claim 10, wherein dynamicallyadding processing units to the data warehouse based on the workloadcomprises: determining whether a query can be processed while meeting aperformance metric for the query; and triggering startup of a newprocessing unit in response to determining that the query in combinationwith a current workload does not allow one or more currently allocatedprocessing units to meet the performance metric.
 13. The system of claim12, wherein the method comprises determining whether the query can beprocessed for each query directed to the processing unit such that theperformance metric is met for each query.
 14. The system of claim 12,wherein the performance metric comprises a maximum time period that thequery will be queued.
 15. The system of claim 10, wherein dynamicallyadding processing units comprises adding processing units up to apredetermined maximum number of processing units.
 16. The system ofclaim 10, wherein dynamically removing processing units comprisesremoving processing units down to a predetermined minimum number ofprocessing units.
 17. The system of claim 10, wherein removingprocessing units based on the workload comprises: determining whether acurrent workload is serviceable by one fewer than the plurality ofprocessing units while meeting a performance metric; and decommissioningat least one processing unit of the plurality of processing units inresponse to determining that the workload is serviceable by one fewerthan the plurality of processing units.
 18. The system of claim 10,wherein providing queries for the data warehouse to each of theplurality of processing units comprises routing queries based on aworkload of each of the plurality of processing units.
 19. Anon-transitory computer-readable medium having instructions storedthereon that, when executed by a processor, cause the processor to:allocate a plurality of processing units as part of a data warehouse,wherein each of at least some of the plurality of processing unitsaccesses data within one or more databases in one or more cloud storageresources; provide, by a processing device, one or more queries to eachprocessing unit within the data warehouse; monitor a workload of theplurality of the processing units to determine that a processingcapacity of the plurality of processing units has reached a thresholdprocessing capacity; and dynamically add additional processing units toand removing processing units from the data warehouse as needed based onthe workload of the processing units indicating that the processingcapacity has reached the threshold processing capacity.
 20. Thenon-transitory computer-readable medium of claim 19, further comprisingdetermining the workload of the processing units, wherein determiningthe workload comprises determining an availability of one or more of:processor resources for each processing unit; and memory resources foreach processing unit.
 21. The non-transitory computer-readable medium ofclaim 19, wherein dynamically adding processing units to the datawarehouse based on the workload comprises: determining whether a querycan be processed while meeting a performance metric for the query; andtriggering startup of a new processing unit in response to determiningthat the query in combination with a current workload does not allow oneor more currently allocated processing units to meet the performancemetric.
 22. The non-transitory computer-readable medium of claim 21,wherein the method comprises determining whether the query can beprocessed for each query directed to the processing unit such that theperformance metric is met for each query.
 23. The non-transitorycomputer-readable medium of claim 21, wherein the performance metriccomprises a maximum time period that the query will be queued.
 24. Thenon-transitory computer-readable medium of claim 19, wherein dynamicallyadding processing units comprises adding processing units up to apredetermined maximum number of processing units.
 25. The non-transitorycomputer-readable medium of claim 19, wherein dynamically removingprocessing units comprises removing processing units down to apredetermined minimum number of processing units.
 26. The non-transitorycomputer-readable medium of claim 19, wherein removing processing unitsbased on the workload comprises: determining whether a current workloadis serviceable by one fewer than the plurality of processing units whilemeeting a performance metric; and decommissioning at least oneprocessing unit of the plurality of processing units in response todetermining that the workload is serviceable by one fewer than theplurality of processing units.
 27. The non-transitory computer-readablemedium of claim 19, wherein providing queries for the data warehouse toeach of the plurality of processing units comprises routing queriesbased on a workload of each of the plurality of processing units.